Zhuming  Ai,  Mark  A.  Livingston,  and  Jonathan  W.  Decker 
Naval  Research  Laboratory 
4555  Overlook  Ave.  SW,  Washington,  DC  20375 
Phone:  202-767-0371,  202-767-0380 

Email:  zhuming.ai@nrl.navy.mil,  mark.livingston@nrl.navy.mil,  jonathan.decker@nrl.navy.mil 

Mission  Specific  Embedded  Training  Using  Mixed  Reality 


Abstract 

High  risk  special  operations  often  require  that  the 
team  is  trained  in  advance  in  an  environment  simi¬ 
lar  to  the  target  area.  It  would  be  ideal  if  a  replica 
could  be  built  and  the  team  could  be  trained  in 
the  operational  environment.  However,  it  is  usu¬ 
ally  difficult  to  do  so  in  a  timely  manner,  espe¬ 
cially  when  intelligence  constantly  changes.  Tradi¬ 
tionally,  a  scaled-down  model  of  the  target  area  is 
built  instead.  Mixed  reality  based  embedded  train¬ 
ing  with  the  ability  to  rapidly  build  and  improve 
the  model  is  a  useful  tool  that  can  meet  these  chal¬ 
lenges. 

Several  research  groups  have  considered  aug¬ 
mented  reality  (AR)  or  mixed  reality  as  a  training 
tool  for  military  operations  in  urban  terrain.  Our 
group  has  developed  the  Battlefield  Augmented 
Reality  System  (BARS™),  which  can  be  used  for  a 
variety  of  applications,  such  as  situation  awareness 
as  well  as  embedded  training.  We  have  since  de¬ 
veloped  a  new  version  of  the  system  that  makes  use 
of  the  state-of-the-art  techniques  to  build  mission- 
specific  training  systems  in  a  timely  manner.  While 
there  is  still  debate  on  how  effective  a  current  AR 
situation  awareness  system  is  in  an  actual  combat 
situation,  we  have  seen  increased  interest  in  AR 
training  among  Marines,  Air  Force  special  forces, 
and  others. 

In  this  paper,  a  mission  specific  embedded  mil¬ 
itary  training  system  using  mixed  reality  is  pre¬ 
sented.  The  methods  for  building  environmental 
models  and  embedding  synthetic  characters  are  de¬ 
scribed.  A  set  of  tools  has  been  assembled  to  build 
mission  specific  models.  These  models  together 
with  a  modular  physical  setting  can  be  integrated 


into  the  training  system  which  provides  a  mixture 
of  virtual  objects  and  physical  objects.  This  mixed 
environment  provides  a  more  realistic  training  than 
a  pure  virtual  environment,  and  more  flexible  train¬ 
ing  than  pure  physical  settings. 

1  Introduction 

To  ensure  the  success  of  high  risk  special  opera¬ 
tions,  it  is  often  required  that  the  special  operation 
team  is  trained  in  advance  in  a  environment  similar 
to  the  target  area.  It  would  be  ideal  if  a  physical 
replica  could  be  built  and  the  team  could  be  trained 
in  the  environment.  However,  it  is  usually  difficult 
to  do  so  in  a  timely  manner,  especially  when  the 
intelligence  is  constantly  changing.  Traditionally 
a  scaled  down  model  is  built  instead.  This  has  a 
negative  impact  on  planning  and  training.  A  mixed 
reality  (MR)  based  embedded  training  system  with 
the  ability  to  rapidly  build  and  improve  the  model 
is  proposed  in  this  paper  to  meet  these  challenges. 

Several  groups  have  considered  augmented  re¬ 
ality  (AR)  or  mixed  reality  as  a  training  tool  for 
military  operations  in  urban  terrain  [1,  2].  Our 
group  has  developed  a  system,  the  Battlefield  Aug¬ 
mented  Reality  System  (BARS™)  [3],  that  can  be 
used  for  a  variety  of  applications,  such  as  situation 
awareness  [4]  as  well  as  embedded  training  [5].  We 
have  since  developed  a  new  version  of  the  system 
that  makes  use  of  the  state-of-the-art  technique  to 
build  mission  specific  training  systems  in  a  timely 
manner.  With  the  ability  to  build  and  update  the 
model  of  the  target  area  within  hours,  the  system 
is  a  useful  tool  for  the  training  of  special  opera¬ 
tions.  While  there  is  still  debate  on  how  effective  a 
current  AR  situation  awareness  system  is  in  an  ac- 
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tual  combat  situation,  we  have  seen  a  lot  of  interest 
in  AR  training  among  Marines,  Air  Force  special 
forces,  and  so  on. 

Among  the  most  recent  efforts  on  AR/MR  mil¬ 
itary  applications,  the  DARPA  sponsored  Urban 
Leader  Tactical  Response,  Awareness  and  Visu¬ 
alization  (ULTRA-Vis)  project  is  targeted  to  giv¬ 
ing  small  unit  leaders  the  capability  to  issue  com¬ 
mands  and  share  actionable  mission-relevant  in¬ 
formation  in  an  urban  environment  non-line-of- 
sight.  [6]  AR/MR  has  also  found  other  applications 
in  military  training,  such  as  a  virtual  sand  table[7], 
armored  personnel  carrier  turret  maintenance  [8], 
and  infantry  skills  training  [9]. 

In  this  paper,  a  mission  specific  embedded  mil¬ 
itary  training  system  using  mixed  reality  is  pre¬ 
sented.  The  methods  for  building  environmental 
models  and  embedding  synthetic  characters  are  de¬ 
scribed. 

2  Methods 

The  mixed  reality  based  mission  specific  embed¬ 
ded  training  system  is  developed  on  top  of  a  gen¬ 
eral  AR  embedded  training  system.  An  MR  system 
is  a  complex  system  that  has  many  hardware  and 
software  components:  a  tracking  component,  a  dis¬ 
play  component,  user  interface,  world  model,  etc. 
In  addition  to  the  components  that  are  common  in 
mixed  reality  training  systems,  the  ability  to  create 
mission  specific  environmental  models  rapidly  and 
insert  synthetic  characters  based  on  requirements 
of  the  mission  are  two  main  aspects  that  are  unique 
to  BARS™. 

A  set  of  tools  has  been  assembled  in  BARS™to 
build  mission  specific  models.  These  models,  to¬ 
gether  with  a  modular  physical  setting,  can  be  in¬ 
tegrated  into  the  system  which  provides  a  mixture 
of  virtual  objects  and  physical  objects.  This  mixed 
environment  provides  a  more  realistic  training  than 
a  pure  virtual  environment,  and  more  flexible  train¬ 
ing  than  pure  physical  settings.  Synthetic  charac¬ 
ters  are  included  in  the  system;  their  behavior  can 
be  programmed  to  mimic  the  enemy  forces  or  other 
entities  that  are  more  convenient  to  control  digi¬ 
tally.  For  example,  the  system  can  be  used  to  train 
how  to  behave  in  a  manner  compatible  with  cultural 
norms.  The  actors  could  become  more  cooperative 


as  a  reward  for  proper  behavior  or  more  hostile  as 
a  cost  of  inappropriate  behavior. 

2.1  Environmental  Modeling 

The  environmental  models  are  built  using  data  from 
various  sources,  depending  on  the  availability  of 
the  intelligence.  In  general,  three  different  source 
data  are  used  in  our  study.  Digital  elevation  maps 
are  used  to  define  the  shape  of  the  terrain,  images 
that  show  the  texture  of  the  ground  are  used  to  ren¬ 
der  the  background,  and  building  maps  are  used  to 
define  the  shape  of  the  buildings.  In  some  applica¬ 
tions,  detailed  building  models  may  be  required. 

Satellite  images  and  digital  elevation  data  are 
used  to  build  the  terrain  model.  Light  Detection 
And  Ranging  (LIDAR)  data  of  the  buildings  are 
used  as  initial  template  to  build  models  of  the  build¬ 
ings.  These  geo-referenced  data  sets  are  automati¬ 
cally  or  semi- automatically  integrated,  while  other 
details  are  added  manually  based  either  on  intelli¬ 
gence  or  assumptions. 

The  modeling  workflow  is  shown  in  Figure  1  and 
depicts  the  environmental  model.  The  environmen¬ 
tal  model  consists  of  a  terrain  model  and  a  building 
model.  The  terrain  model  is  built  based  on  digi¬ 
tal  elevation  data  (Figure  1  (a)),  which  usually  is 
a  GeoTIFF  file,  and  satellite  images  (Figure  1  (b)) 
that  are  used  as  textures. 

Several  open  source  or  freely  available  software 
tools  are  used  to  create  the  terrain  model.  Usually 
some  artifacts  are  present  in  the  digital  elevation 
GeoTIFF  file,  and  a  pre-processing  step  is  needed. 
FWTools[10]  is  used  for  pre-processing,  includ¬ 
ing  cropping,  geo-information  extraction,  etc.  It  is 
also  used  to  extract  geo-tag  information  that  will  be 
used  in  further  steps. 

Satellite  images  or  aerial  photo  images  are  useful 
when  creating  the  texture  of  the  ground.  One  con¬ 
venient  source  of  satellite  images  is  Google  Earth 
or  Google  Maps.  These  satellite  images  are  used 
as  texture  maps  in  building  the  terrain  model.  To 
extract  satellite  images  from  Google  Earth,  place- 
markers  are  placed  at  the  four  corners  with  the 
same  coordinates  of  the  GeoTIEE  file  of  the  digi¬ 
tal  elevation  data.  Google  Earth  only  allows  input 
of  latitude/longitude  data;  thus,  coordinate  system 
conversion  is  needed.  The  captured  images  need 
to  be  cropped  using  image  processing  software.  To 
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(c)  Building  shapefile.  (d)  3D  model  in  the  training  environment. 


Figure  1 :  Modeling  the  Environment. 


capture  high-resolution  texture  images,  some  kind 
of  stitching  methods  need  to  be  used.  One  pos¬ 
sibility  is  to  use  Google  Maps  API  to  get  high- 
resolution  images. 

The  geo-referenced  information  in  the  digital  el¬ 
evation  data  needs  to  be  transferred  to  the  texture 
map.  This  can  make  sure  that  the  two  data  will 
align  with  each  other.  Satellite  images  and  digital 
elevation  data  are  combined  using  an  open  source 
tool  called  VirtualPlanetBuilder[ll]. 

Compared  to  environmental  models,  the  process 
of  creating  building  models  is  more  complicated. 
Although  tools  such  as  VirtualPlanetBuilder  should 
have  the  ability  to  create  models  from  a  shapefile. 


the  functions  are  often  not  working  well.  A  differ¬ 
ent  set  of  tools  are  used  in  our  study.  The  shapefile 
(Figure  1  (c))  is  first  converted  to  a  3D  model  using 
a  geographic  information  system  (CIS)  software. 
It  is  important  to  make  sure  that  the  base  heights 
match  the  digital  elevation  data.  The  base  heights 
information  in  the  shapefile  may  not  agree  with  the 
information  in  the  elevation  file. 

The  created  3D  model  file  can  be  imported  into 
a  3D  modeling  software  to  do  modification,  texture 
mapping,  and  adding  other  objects,  etc. 

The  terrain  and  the  building  are  combined  into 
the  MR  simulation  environment  (Figure  1  (d)).  It 
is  very  important  to  keep  the  alignment  among  dif- 
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(a)  Synthetic  characters  in  an  indoor  environment.  (h)  Synthetic  characters  in  an  outdoor  environment. 

Figure  2:  Embedded  training  with  augmented  reality. 


ferent  models.  The  terrain  model  is  in  the  coordi¬ 
nate  system  that  was  defined  in  the  geo-tags.  When 
the  building  model  is  created  from  the  shapefile,  it 
might  be  in  a  different  coordinate  system.  The  off¬ 
set  between  the  building  and  the  terrain  should  also 
be  calculated. 

2.2  Actor  Modeling  and  Artificial  Intelli¬ 
gence 

Opposing  forces  can  also  be  digitally  built  in  the 
system.  The  behavior  of  the  synthetic  characters 
in  this  training  environment  is  controlled  by  an  ar¬ 
tificial  intelligence  (AI)  module.  It  is  further  pro¬ 
grammed  with  a  scripting  language. 

The  intelligence  of  the  synthetic  actors  are  de¬ 
signed  in  two  levels.  First,  a  navigation  system  is 
implemented  so  that  the  actors  can  navigate  auto¬ 
matically  in  the  scene  when  a  target  position  is  des¬ 
ignated.  Secondly,  the  behavior  of  the  characters  is 
controlled  by  a  two-stage  state  machine. 

The  top  level  state  machine  works  at  mission 
level.  It  controls  how  a  character  acts  when  it  re¬ 
ceives  a  certain  command.  It  also  controls  the  re¬ 
sponse  when  it  encounters  a  certain  event.  For  ex¬ 
ample,  when  a  soldier  actor  hears  an  explosion,  the 
state  machine  can  decide  to  send  it  to  investigate 
the  incident  or  search  for  cover.  In  each  state  in  this 
top  level  state  machine,  there  is  a  second  state  ma¬ 
chine.  This  secondary  state  machine  controls  the 
low  level  animation  sequence  of  the  character. 

Figure  2  shows  the  synthetic  actors  in  the  MR 
environments,  indoor  and  outdoor. 


Two  approaches  have  been  implemented  to  in¬ 
clude  animated  characters  into  the  system.  One 
is  using  DI-Guy  software  development  kit,  which 
is  developed  by  Boston  Dynamics.  The  other  ap¬ 
proach  is  an  open  source  approach.  It  uses  a  3D 
character  animation  library,  Cal3D.[12] 

BARS™can  also  connect  to  the  US  Army’s 
OneSAF  Testbed  Baseline  Semi-Automated  Forces 
(0TB  SAF)  system  to  use  computer-generated 
forces.  The  system  connects  to  a  local  instance 
of  the  Run-Time  Infrastructure  (RTI)  which  0TB- 
SAF  also  connects  through  a  gateway.  The  users 
are  reflected  in  real  time  in  OTBSAF  as  friendly 
forces,  and  the  computer-generated  forces  respond 
appropriately.  These  responses  are  sent  to  the 
training  system  to  control  the  visualizations  of  the 
computer-generated  forces.  [2] 

2.3  Implementation 

BARS™includes  a  tracking  component,  a  display 
component,  a  network  component,  a  database  com¬ 
ponent,  and  an  AFanimation  component  (Figure  3). 
The  tracking  component  is  a  generalized  driver  that 
supports  a  variety  of  hardware  sensors,  such  as 
GPS,  inertial,  acoustic,  optical  sensors,  etc.  These 
sensors  can  be  dynamically  assigned  to  track  the 
position  and  orientation  of  the  user  and/or  the  po¬ 
sition  and  orientation  of  a  weapon.  The  display 
component  supports  head-worn-display,  both  op¬ 
tical  see-through  or  video  see-through.  The  net¬ 
work  component  allows  multi-user  training  in  a 
networked  environment.  The  database  component 
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Figure  3:  BARS  system  diagram. 

ean  import  models  from  different  sourees. 

The  software  system  is  implemented  around  a 
open  souree  game  engine,  Delta3D[13],  whieh 
uses  a  OpenGL  based  open  souree  toolkit, 
OpenSeeneGraph[14]  for  rendering.  A  mobile  ver¬ 
sion  of  the  system  makes  use  of  portable  eomputers 
to  assemble  the  system  into  a  baekpaek.  A  wireless 
network  provides  eommunieation  among  different 
mobile  systems. 

3  Conclusion 

This  paper  deseribed  a  mixed  reality  method  for 
military  training  that  ineorporates  mission  speeifie 
environmental  models  and  digital  opposing  forees 
and  other  synthetie  eharaeters.  A  workflow  has 
been  developed  to  ereate  the  environmental  model 
quiekly  and  ean  be  modified  easily  based  on  the 
eonstantly  ehanging  intelligenee.  These  models  to¬ 
gether  with  a  modular  physieal  setting  ean  be  in¬ 
tegrated  into  a  system  that  provides  a  mixture  of 
virtual  objeets  and  physieal  objeets.  This  mixed  en¬ 
vironment  provides  a  more  realistie  training  than  a 
pure  virtual  environment,  and  more  flexible  train¬ 
ing  than  pure  physieal  settings. 
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